Repository navigation
feat(review): incremental and snapshot semantics that fit any dbt project - #1725
RaghwendraSingh wants to merge 6 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
| // History diverges from new rows when the incremental filter changes or a | ||
| // column is added (old rows get NULL). Formatting-only edits never count, | ||
| // and `on_schema_change='fail'` has its own finding below. | ||
| if (!isNew && input.oldSql && cfg.onSchemaChange !== "fail" && bodyTokens(input.oldSql) !== bodyTokens(input.newSql) && |
There was a problem hiding this comment.
WARNING: on_schema_change='fail' also hides is_incremental() filter changes
The cfg.onSchemaChange !== "fail" guard skips this whole block before filterChanged is computed, and the on_schema_change finding below only fires when columnDelta() reports added/removed columns. So a model with on_schema_change='fail' whose PR changes only the incremental filter (no column change) gets no finding from either rule — the exact history divergence this rule targets. Only defer when the deferral actually applies, e.g. if ((cfg.onSchemaChange !== "fail" || added.length) && (filterChanged || added.length)).
There was a problem hiding this comment.
Fixed in 7a61dd8. Under on_schema_change='fail' only column changes go to the fail finding, and a change to the is_incremental() filter alone is reported as needing a history plan again. There is a test for it.
| const prev = before.get(name) | ||
| if (!prev) continue | ||
| const changed = ["check_cols", "strategy"].filter( | ||
| (k) => prev[k] !== undefined && JSON.stringify(prev[k]) !== JSON.stringify(cfg[k]), |
There was a problem hiding this comment.
SUGGESTION: Semantics-preserving check_cols edits are reported as a change
JSON.stringify compares arrays positionally, so reordering check_cols ([status, amount] -> [amount, status]) reports a check_cols change even though the snapshot tracks the same set. Normalize before comparing (sort the array values) here and in the norm() comparison in snapshotFindings (line 297) so a reorder stays silent.
There was a problem hiding this comment.
Follow-up: low priority, tracked. We'll sort check_cols before comparing.
| out.push( | ||
| finding({ file, model: name }, "snapshot-tracking-change", "warning", "materialization", | ||
| `snapshot ${changed.map((k) => `\`${k}\``).join(" and ")} changed`, | ||
| "Changing what an existing snapshot compares can version every row on the next run (a column new to the snapshot compares " + |
There was a problem hiding this comment.
SUGGESTION: Duplicated snapshot-change body and key list
The check_cols/strategy key list and the whole "Changing what an existing snapshot compares…" body are copied verbatim from snapshotFindings (lines 297-303). Extract a single helper (e.g. snapshotTrackingFinding(file, model, before, after)) so the SQL and yml snapshot paths cannot drift in wording or thresholds.
There was a problem hiding this comment.
Follow-up: nit, tracked. We'll dedupe the snapshot key list and body.
| * whose first capture group is the list, `model` the model it mirrors | ||
| * (omitted = files that mention the model by name). | ||
| */ | ||
| /** |
There was a problem hiding this comment.
SUGGESTION: Inserted JSDoc now documents the wrong field
This block landed between the existing consumerColumnLists doc comment (lines 44-49) and the field it documents, so consumerColumnLists is left undocumented and its "Opt-in: explicit column lists outside dbt…" text now reads as if it describes incrementalSliceVars. Move incrementalSliceVars above that comment, or move the consumerColumnLists comment back beside its field.
There was a problem hiding this comment.
Follow-up: nit, tracked. The JSDoc will move back onto incrementalSliceVars.
| import { capabilityFor } from "./adapter-capabilities" | ||
|
|
||
| /** | ||
| * dbt incremental, microbatch and snapshot semantics. These rules follow |
There was a problem hiding this comment.
SUGGESTION: Module doc still mentions microbatch
The microbatch rule and its capability entry were removed in this PR, so "incremental, microbatch and snapshot semantics" no longer matches what the file implements. Drop "microbatch".
There was a problem hiding this comment.
Follow-up: nit, tracked. We'll drop "microbatch" from the module doc.
Code Review SummaryStatus: 8 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (10 files in the incremental diff)
Fix these issues in Kilo Cloud Previous Review Summary (commit fe31dc6)Current summary above is authoritative. Previous snapshots are kept for context only. Previous review (commit fe31dc6)Status: 5 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)WARNING
SUGGESTION
Files Reviewed (13 files)
Reviewed by deepseek-v4.1-flash-1017 · Input: 158.1K · Output: 42.3K · Cached: 2.6M Review guidance: REVIEW.md from base branch |
There was a problem hiding this comment.
13 issues found across 13 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/src/altimate/review/adapter-capabilities.ts">
<violation number="1" location="packages/opencode/src/altimate/review/adapter-capabilities.ts:41">
P3: These source links point at the `master` branch of dbt-trino, which moves over time. Pin them to a release tag or commit SHA so the cited behavior stays verifiable.</violation>
</file>
<file name="docs/docs/usage/dbt-pr-review.md">
<violation number="1" location="docs/docs/usage/dbt-pr-review.md:507">
P2: The detector recognizes a listed `var()` only on the right side of `=` or `IN`; range and reversed comparisons are missed. Limit the documentation to supported filters or extend detection.</violation>
<violation number="2" location="docs/docs/usage/dbt-pr-review.md:514">
P2: `DBT011`/`DBT012` don't appear anywhere in the repo, so this sentence points readers at rule IDs they can't find. Name the rule that actually covers `unique_key` and strategy changes, or remove the reference.</violation>
</file>
<file name="packages/opencode/src/altimate/review/incremental-semantics.ts">
<violation number="1" location="packages/opencode/src/altimate/review/incremental-semantics.ts:145">
P2: The non-greedy match stops at the first `endif`, not the one closing `is_incremental()`. With a nested Jinja `if`, changes after the inner block are omitted from `incrementalFilter` and can suppress the history finding; balance the Jinja blocks.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/incremental-semantics.ts:236">
P2: This skips the case where `partition_by` is added to an existing incremental model. The existing table keeps no partitioning, so that case should warn too. Remove the old-value guard and rely on `configChanged`.</violation>
</file>
<file name="packages/opencode/src/altimate/review/dbt-patterns.ts">
<violation number="1" location="packages/opencode/src/altimate/review/dbt-patterns.ts:262">
P2: This detector only matches a `row_number() over (...)` whose window fits on one added line, so it misses the usual multi-line dbt dedup layout and any window whose PARTITION BY calls a function. Match against the joined added block instead, and use a regex that allows one level of nested parentheses.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/dbt-patterns.ts:262">
P2: `ROW_NUMBER` can merely number rows; this emits a dedup finding even when `rn` is only selected and no rows are filtered, as in the new test. Require evidence that the rank is used to retain one row before reporting that a version survives.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/dbt-patterns.ts:281">
P2: This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a `dedup:load-timestamp-only` entry to `CORE_AST_COVERED` in orchestrate.ts, or drop the QUALIFY form here, so one issue is not posted twice.</violation>
</file>
<file name="packages/opencode/src/altimate/review/rule-catalog.ts">
<violation number="1" location="packages/opencode/src/altimate/review/rule-catalog.ts:171">
P2: Gating `rs-getdate` to Redshift drops the idempotency warning for other engines that support `sysdate()`, such as Snowflake. The rule's own comment says this concern is kept across engines, so keep the rule engine-agnostic by removing the `Redshift:` prefix from its title or excluding it from the engine filter.</violation>
</file>
<file name="packages/opencode/src/altimate/review/orchestrate.ts">
<violation number="1" location="packages/opencode/src/altimate/review/orchestrate.ts:641">
P2: `outputColumns` merges in `selectListNames`, which reads only the last top-level SELECT. For UNION queries the output names come from the first branch, so a last-branch alias change can show up as a removed or added column. The consumer lanes then emit a false `removed column(s)` warning. Skip the merge when the SQL has a set operator, or read the first branch instead, and add a UNION case to the select-list tests.</violation>
</file>
<file name="packages/opencode/src/altimate/review/select-list.ts">
<violation number="1" location="packages/opencode/src/altimate/review/select-list.ts:42">
P2: The parser scans delimiters without respecting double-quoted identifiers: `select 1 as "from" from t` ends the list inside the alias. `blankComments` also strips `--` inside string literals, losing aliased outputs; make comment and keyword scanning quote-aware.
(Based on your team's feedback about quoted identifiers in SELECT scans.)</violation>
<violation number="2" location="packages/opencode/src/altimate/review/select-list.ts:53">
P2: Returning `null` on `*` discards every other aliased expression, even when lineage expands the wildcard. For `select *, 1 as new_col from t`, `outputColumns` keeps the lineage columns but loses `new_col`, so column-delta rules miss it; retain known aliases separately from wildcard uncertainty.</violation>
</file>
<file name="packages/opencode/src/altimate/review/runner.ts">
<violation number="1" location="packages/opencode/src/altimate/review/runner.ts:449">
P2: If packages contain models with the same basename, this returns whichever manifest node appears first. The caller passes only `modelNameFromPath` (the basename), so the rules can use another model's materialization or `unique_key`; resolve the node by path or `unique_id`.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| runs `create or replace` then `insert`, and `drop` removes the table during | ||
| the build, so readers outside dbt can see an empty or missing table (a | ||
| suggestion). `unique_key` and strategy changes are covered by the engine's | ||
| `DBT011`/`DBT012`. |
There was a problem hiding this comment.
P2: DBT011/DBT012 don't appear anywhere in the repo, so this sentence points readers at rule IDs they can't find. Name the rule that actually covers unique_key and strategy changes, or remove the reference.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/docs/usage/dbt-pr-review.md, line 514:
<comment>`DBT011`/`DBT012` don't appear anywhere in the repo, so this sentence points readers at rule IDs they can't find. Name the rule that actually covers `unique_key` and strategy changes, or remove the reference.</comment>
<file context>
@@ -486,6 +487,31 @@ The deterministic rules group into four families:
+ runs `create or replace` then `insert`, and `drop` removes the table during
+ the build, so readers outside dbt can see an empty or missing table (a
+ suggestion). `unique_key` and strategy changes are covered by the engine's
+ `DBT011`/`DBT012`.
- **dbt config / Jinja (`DBT0xx`).** A core **minijinja** parser reads
`{{ config(...) }}` and the model body: incremental model without an
</file context>
There was a problem hiding this comment.
Not changing. These are altimate-core dbt_config_diff rule ids, and the bundled core 0.7.0 ships DBT011 unique_key_change and DBT012 incremental_strategy_change.
| } | ||
| } | ||
|
|
||
| if (configChanged("partition_by") && inlineOld.partition_by !== undefined) |
There was a problem hiding this comment.
P2: This skips the case where partition_by is added to an existing incremental model. The existing table keeps no partitioning, so that case should warn too. Remove the old-value guard and rely on configChanged.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/incremental-semantics.ts, line 236:
<comment>This skips the case where `partition_by` is added to an existing incremental model. The existing table keeps no partitioning, so that case should warn too. Remove the old-value guard and rely on `configChanged`.</comment>
<file context>
@@ -0,0 +1,348 @@
+ }
+ }
+
+ if (configChanged("partition_by") && inlineOld.partition_by !== undefined)
+ out.push(
+ finding(input, "incremental-partition-change", "warning", "materialization", "`partition_by` changed on an incremental model",
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. We'll report a newly added partition_by.
| model: c.model, | ||
| confidence: "high", | ||
| evidence: { tool: "dbt-patterns", result: { rule: "dedup_load_timestamp_only", line: line.trim() } }, | ||
| ruleKey: "dedup:load-timestamp-only", |
There was a problem hiding this comment.
P2: This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a dedup:load-timestamp-only entry to CORE_AST_COVERED in orchestrate.ts, or drop the QUALIFY form here, so one issue is not posted twice.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/dbt-patterns.ts, line 281:
<comment>This load-timestamp dedup finding overlaps with core lint L039, which already flags timestamp-only ROW_NUMBER dedup. Add a `dedup:load-timestamp-only` entry to `CORE_AST_COVERED` in orchestrate.ts, or drop the QUALIFY form here, so one issue is not posted twice.</comment>
<file context>
@@ -249,6 +249,41 @@ const detectDedupTie: Detector = (c) => {
+ model: c.model,
+ confidence: "high",
+ evidence: { tool: "dbt-patterns", result: { rule: "dedup_load_timestamp_only", line: line.trim() } },
+ ruleKey: "dedup:load-timestamp-only",
+ })
+ }
</file context>
There was a problem hiding this comment.
Not changing: the bundled core 0.7.0 has no L039 lint code, so we couldn't confirm an overlap.
| pos += piece.length + 1 | ||
| const item = piece.replace(/^\s*(distinct|all)\s+/i, "").trim() | ||
| if (!item) continue | ||
| if (/(^|\.)\*$/.test(item)) return null |
There was a problem hiding this comment.
P2: Returning null on * discards every other aliased expression, even when lineage expands the wildcard. For select *, 1 as new_col from t, outputColumns keeps the lineage columns but loses new_col, so column-delta rules miss it; retain known aliases separately from wildcard uncertainty.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/select-list.ts, line 53:
<comment>Returning `null` on `*` discards every other aliased expression, even when lineage expands the wildcard. For `select *, 1 as new_col from t`, `outputColumns` keeps the lineage columns but loses `new_col`, so column-delta rules miss it; retain known aliases separately from wildcard uncertainty.</comment>
<file context>
@@ -0,0 +1,67 @@
+ pos += piece.length + 1
+ const item = piece.replace(/^\s*(distinct|all)\s+/i, "").trim()
+ if (!item) continue
+ if (/(^|\.)\*$/.test(item)) return null
+ const rawItem = raw.trim().replace(/^(distinct|all)\s+/i, "")
+ const bare = /^(?:[A-Za-z_][A-Za-z0-9_$]*\.)*("[^"]+"|[A-Za-z_][A-Za-z0-9_$]*)$/.exec(rawItem)
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. We'll keep known aliases next to a *.
| try { | ||
| const raw = loadRawManifest(opts.manifestPath) | ||
| const hit = Object.entries<any>(raw?.nodes ?? {}).find( | ||
| ([id, n]) => (id.startsWith("model.") || id.startsWith("snapshot.")) && n?.name === name, |
There was a problem hiding this comment.
P2: If packages contain models with the same basename, this returns whichever manifest node appears first. The caller passes only modelNameFromPath (the basename), so the rules can use another model's materialization or unique_key; resolve the node by path or unique_id.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/runner.ts, line 449:
<comment>If packages contain models with the same basename, this returns whichever manifest node appears first. The caller passes only `modelNameFromPath` (the basename), so the rules can use another model's materialization or `unique_key`; resolve the node by path or `unique_id`.</comment>
<file context>
@@ -442,6 +442,19 @@ export function createDispatcherRunner(opts: DispatcherRunnerOptions): ReviewRun
+ try {
+ const raw = loadRawManifest(opts.manifestPath)
+ const hit = Object.entries<any>(raw?.nodes ?? {}).find(
+ ([id, n]) => (id.startsWith("model.") || id.startsWith("snapshot.")) && n?.name === name,
+ )
+ return hit?.[1]?.config as Record<string, unknown> | undefined
</file context>
There was a problem hiding this comment.
Follow-up: low priority, tracked. We'll look up by unique_id. Same as #1724 r4626975851.
| "https://trino.io/docs/current/functions/math.html", | ||
| // create_table_as with an enforced contract: `create or replace table` then `insert`; | ||
| // on_table_exists='drop' drops the table before building it. | ||
| "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/adapters.sql", |
There was a problem hiding this comment.
P3: These source links point at the master branch of dbt-trino, which moves over time. Pin them to a release tag or commit SHA so the cited behavior stays verifiable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/opencode/src/altimate/review/adapter-capabilities.ts, line 41:
<comment>These source links point at the `master` branch of dbt-trino, which moves over time. Pin them to a release tag or commit SHA so the cited behavior stays verifiable.</comment>
<file context>
@@ -26,7 +33,14 @@ export const ADAPTER_CAPABILITIES: Record<string, AdapterCapability> = {
+ "https://trino.io/docs/current/functions/math.html",
+ // create_table_as with an enforced contract: `create or replace table` then `insert`;
+ // on_table_exists='drop' drops the table before building it.
+ "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/adapters.sql",
+ "https://github.com/starburstdata/dbt-trino/blob/master/dbt/include/trino/macros/materializations/table.sql",
+ ],
</file context>
There was a problem hiding this comment.
Follow-up: nit, tracked. We'll pin the dbt-trino links.
hkaltai
left a comment
There was a problem hiding this comment.
Panel: Claude, Codex, and lead reviewer. No CRITICAL or MAJOR findings; all findings are advisory-level correctness gaps. Counts: CRITICAL 0, MAJOR 0, MINOR 6 (inline), NIT 9.
The new rules are well scoped: adapter-specific behaviour is capability-gated, the slice rule is opt-in, formatting-only edits are filtered, and findings cannot block a merge. The main accuracy gaps are (1) Spark: dialect-pack rules remain ungated, (2) change detection compares only inline config, so moving config to yml produces false positives, and (3) the select-list reader mishandles UNION branches and -- inside string literals. Tests could not be fully run in the review environment (missing dependencies); the select-list tests pass. Missing tests: UNION output names, -- in literals, inline-to-yml config moves, Spark rules under another dialect.
Nits
packages/opencode/src/altimate/review/config.ts:44-55:incrementalSliceVarsand its JSDoc sit betweenconsumerColumnLists's JSDoc and the field, leaving that JSDoc dangling. Move the new field afterconsumerColumnLists.incremental-semantics.ts:7andorchestrate.ts:785: comments say "microbatch" but no microbatch rule remains. Drop the word.incremental-semantics.ts:239:"The existing table keeps its old partitioning"understates it; on BigQuery/Spark the next run can fail. Suggest "...the next incremental run fails or keeps the old partitioning, depending on the adapter."incremental-semantics.ts:283-285: both branches of thehasExposuresternary say nearly the same thing; simplify.orchestrate.ts:639: comment "*in the final SELECT: keep lineage's own answer" sits above code with no separate branch; reword or remove.contractTableReplaceAtomicalso governson_table_exists='drop', which has nothing to do with contracts; considertableRebuildAtomic.ticksis defined in bothorchestrate.ts:666andincremental-semantics.ts:204; share one.incrementalSemanticsLanecallscolumnDeltaagain; two consumer lanes call it too, so lineage can run up to three times per model. Memoize per model context.- Also consider: per-slice
unique_keyrule requires an explicitincremental_strategy; with the adapter default (merge/delete+insert) it is silent (opt-in, so a gap not a bug). Normalizing SQL lowercases string literal values, so'paid'->'PAID'produces no history-plan finding.
|
|
||
| /** The engine a dialect-pack rule is about (from its `Engine:` title prefix), if any. */ | ||
| export function ruleEngine(rule: Rule): string | undefined { | ||
| const m = /^(BigQuery|Snowflake|Redshift|Databricks):/.exec(rule.title) |
There was a problem hiding this comment.
[MINOR] Dialect gating skips the Spark: rules
const m = /^(BigQuery|Snowflake|Redshift|Databricks):/.exec(rule.title)
Three rules are titled Spark: ... (collect_list/collect_set ordering, CACHE/UNCACHE TABLE, to_date without format) and Spark/Databricks: explode() has a different prefix; none match, so ruleEngine returns undefined and they fire on every engine. On a Snowflake or Postgres project, to_date(order_date) still triggers "Spark: to_date/to_timestamp without format", which is the exact problem this gating is meant to fix.
Fix: add Spark (and Spark/Databricks) to the regex and map them to databricks, or store the engine explicitly on each pack rule instead of parsing the title. Add a test for a Spark rule under a non-Databricks dialect.
There was a problem hiding this comment.
Good catch, thanks. Fixed in fix(review): address review feedback: Spark: gating, quote-aware comments, effective-config changes, lineage-less column sets: Spark: and Spark/Databricks: titles map to databricks (a spark dialect counts as Databricks), so to_date(order_date) no longer fires on Snowflake or Postgres. Tested under Snowflake, Postgres, Databricks, Spark and an unknown engine.
| const flat = flatten(sql) | ||
| const selects = [...flat.matchAll(/\bselect\b/gi)] | ||
| if (!selects.length) return null | ||
| const start = selects[selects.length - 1].index! + "select".length |
There was a problem hiding this comment.
[MINOR] selectListNames reads the last branch of a UNION
const start = selects[selects.length - 1].index! + "select".length
For select id, amount as revenue from a union all select id, total from b, output columns are named by the first branch, but this returns the last branch's names (id, total). outputColumns merges them with lineage, so columnDelta can report a column added or removed that the model never exposes. Appending union all select id, total from c yields a bogus "total is new" history-plan suggestion and consumer-lane warnings.
Fix: return null when the flattened text has a top-level union|except|intersect, or take the select list from the first branch.
There was a problem hiding this comment.
Thanks. Fixed in fix(review): address review comments: filter-only change under on_schema_change='fail', UNION output names: a top-level set operation takes its names from the first branch. Tested.
|
|
||
| /** Text with comments blanked, same length. */ | ||
| function blankComments(sql: string): string { | ||
| return sql.replace(/--[^\n]*/g, (m) => " ".repeat(m.length)).replace(/\/\*[\s\S]*?\*\//g, (m) => " ".repeat(m.length)) |
There was a problem hiding this comment.
[MINOR] Comment stripping runs before string handling
return sql.replace(/--[^\n]*/g, (m) => " ".repeat(m.length))...
A -- inside a string literal truncates the rest of the line: select id, 'x--y' as tag, z from t yields ["id"], so tag and z are missed (and lineage-reported columns can then look removed). modelBody in incremental-semantics.ts has the same pattern, so an edit after 'a--b' on the same line is invisible to the history rules.
Fix: use one scanner that skips quoted strings before recognising comments (closing() already tracks quote state).
There was a problem hiding this comment.
Agreed, thanks. Fixed in the new commit: one quote-aware blankComments scanner skips strings and quoted identifiers before recognising -- and /* */. selectListNames and modelBody both use it, so select id, 'x--y' as tag, z gives all three names and an edit after 'a--b' counts. modelBody also keeps string literals' case now, so 'paid' → 'PAID' is a change (your last nit).
| } | ||
| } | ||
|
|
||
| if (configChanged("partition_by") && inlineOld.partition_by !== undefined) |
There was a problem hiding this comment.
[MINOR] partition_by change compares only inline config
if (configChanged("partition_by") && inlineOld.partition_by !== undefined)
Moving partition_by from inline config() into schema.yml or dbt_project.yml leaves the effective value unchanged, but old-inline vs new-inline differs, so the author gets a "run --full-refresh" warning that is not needed. Conversely, a real change made in yml, or a new inline override of an inherited value, is never seen. Verified by reading configChanged and the inlineOld guard.
Fix: compare the old effective value (manifest base overlaid by old inline) against the new effective value, or at least stay silent when the new inline value is missing and the manifest value matches the old inline one.
There was a problem hiding this comment.
Thanks. Fixed in the new commit, partly. An inline value that is removed while the manifest carries the same value (moved to yml or dbt_project.yml) is no longer reported, and a removal with no yml value still is. A new inline override of an inherited value stays silent: without the base revision's effective config we can't tell whether the value changed. A real change made only in yml isn't seen by this rule yet. Both are tracked follow-ups.
| const before = inlineConfig(input.oldSql) | ||
| const after = inlineConfig(input.newSql) | ||
| const out: Finding[] = [] | ||
| const changed = ["check_cols", "strategy"].filter((k) => before[k] !== undefined && norm(before[k]) !== norm(after[k])) |
There was a problem hiding this comment.
[MINOR] Snapshot tracking change has the same inline-only comparison
const changed = ["check_cols", "strategy"].filter((k) => before[k] !== undefined && norm(before[k]) !== norm(after[k]))
Moving strategy or check_cols out of the inline config into yml reports a tracking change although the effective config is identical; adding an inline override to a snapshot that inherited its config is not reported.
Fix: compare effective tracking configs on both revisions.
There was a problem hiding this comment.
Thanks. Fixed in the new commit the same way: snapshots now receive the manifest config, and moving check_cols/strategy to yml with the same value is not a tracking change. Tested both ways.
| if (cols.some((c) => c === "*" || c.endsWith(".*"))) return new Set() | ||
| return new Set(cols) | ||
| // `*` in the final SELECT: keep lineage's own answer (it expands stars it can resolve). | ||
| const named = selectListNames(sql) |
There was a problem hiding this comment.
[MINOR] Select-list names fill in when lineage returned nothing
return new Set([...cols, ...(named ?? [])])
Previously an empty lineage result gave an empty set and columnDelta reported unknown. Now select-list names fill in, skipping unaliased expressions. If lineage fails on only one revision, every unaliased expression column looks removed, feeding the consumer-contract and consumer-list warnings.
Fix: add select-list names only when lineage returned columns, or pass a flag saying whether lineage succeeded.
There was a problem hiding this comment.
Agreed, thanks. Fixed in the new commit: when lineage returns no columns for a side, that side's set is unknown and select-list names only add to lineage's columns. There's a test where lineage fails on the base only and no column looks removed.
…apter-dependent rules read from the capability table
- `incremental-semantics.ts`: config from the manifest (dbt-merged) overlaid by the head's inline `{{ config() }}`; a new `modelConfig` runner method
- incremental model or snapshot logic changed with no stated plan for history (`--full-refresh`, backfill, forward-only) in the PR
- `partition_by` changed on an incremental model
- `on_schema_change='fail'` with added or removed output columns
- `delete+insert`/`merge` whose `unique_key` lacks the column a `var()` filter slices runs by (one tenant's run deleting another's rows)
- snapshot `check_cols`/`strategy` change, and snapshot query change without a plan
- `row_number()` dedup ordered only by a load timestamp
- capability table: Trino/Athena concurrent Iceberg commits conflict (microbatch without `concurrent_batches=false`); Trino `on_table_exists='replace'` + enforced contract is create-then-insert and `drop` removes the table during the build (warning when the model has consumers); sources cite dbt-trino and Iceberg docs
- all findings are advisory (catalog origin); unchanged real-world corpus: 15/15 caught, 0/5 false positives
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the local end-to-end run on a real DuckDB dbt project: - BigQuery/Snowflake/Redshift/Databricks pack rules (titled `Engine: …`) fire only on that engine once the project's engine is known; an unknown engine keeps today's behavior. `ruleEngine()` maps a rule's title prefix to its engine. - Tests: a DuckDB or Snowflake project gets no BigQuery pack advice; a BigQuery project, or an unknown engine, keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ory rule, opt-in slice vars, no microbatch guess, yml snapshots From an adversarial generality review: - the history rule fires only when an incremental model gains a column or its `is_incremental()` filter changes, compares tokens (formatting never counts), defers to the `on_schema_change='fail'` finding, and is a suggestion - the slice-column `unique_key` rule is opt-in: only vars listed in `incrementalSliceVars` count (a `reporting_currency` var filter is not a slice) - the Trino/Athena microbatch rule and its capability entry are removed: dbt-trino's concurrency default is changing and the connector/table format isn't knowable from the adapter - the non-atomic Trino rebuild is always a suggestion and speaks to readers outside dbt, not downstream dbt models - load-timestamp dedup ignores loader sync columns (`_fivetran_synced`, `_airbyte_*`, `_sdc_*`) and is a suggestion - snapshot `check_cols`/`strategy` changes are also caught in dbt 1.9+ yml snapshots; snapshot query changes compare tokens Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by the local end-to-end run: column lineage only reports columns that read a source column, so `current_date as load_date` or `1 as one` were invisible and adding one under `on_schema_change='fail'` went unreported. Output columns now add the final SELECT's own names (`select-list.ts`: an item counts only when aliased after a complete expression or a bare column; `*` defers to lineage). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hema_change='fail'`, UNION output names - `incremental-semantics.ts`: under `on_schema_change='fail'` only column changes are deferred to the fail finding; a change to the `is_incremental()` filter alone is reported as needing a history plan again - `select-list.ts`: a top-level set operation (`UNION`/`INTERSECT`/`EXCEPT`/ `MINUS`) takes its output names from the first branch, so an alias change in a later branch no longer reads as added or removed columns - tests for both Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mments, effective-config changes, lineage-less column sets - Dialect packs: `Spark:` and `Spark/Databricks:` rules are gated like the other packs and apply to Databricks/Spark only once the engine is known. - `blankComments` skips quoted strings and identifiers before recognising `--` and `/* */`, so `'x--y'` no longer truncates the select list; `modelBody` uses it too and keeps string literals' case, so an edit after `'a--b'` or `'paid'` -> `'PAID'` counts as a change. - `partition_by` and snapshot `check_cols`/`strategy`: an inline value removed while the manifest carries the same value (moved to yml or `dbt_project.yml`) is not reported; snapshots now receive the manifest config. - `outputColumns`: when lineage returns nothing for a side, the column set is unknown; select-list names only add to lineage's columns. - Nits: `partition_by` message says the next run can fail; no "microbatch" in comments; `consumerColumnLists` JSDoc back above its field; one exposures sentence. - Tests: Spark rules under other dialects, `--` in literals, literal case, inline-to-yml moves for `partition_by` and `check_cols`, one-sided lineage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1522b5a to
9fa035e
Compare
fe31dc6 to
9625cfd
Compare
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
2 similar comments
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
👋 This PR was automatically closed by our quality checks. Common reasons:
If you believe this was a mistake, please open an issue explaining your intended contribution and a maintainer will help you. |
|
@hkaltai, re your review: Thanks for the approval and the detailed notes. One commit on top:
|
| const n = after[key] | ||
| if (o === undefined || norm(o) === norm(n)) return false | ||
| const m = manifest?.[key] | ||
| if (n === undefined && m !== undefined && m !== null && canonValue(inlineValue(o)) === canonValue(m)) return false |
There was a problem hiding this comment.
WARNING: Manifest-based suppression can hide a real config change when the manifest is stale
manifestConfig is read from the on-disk head manifest, which this review already knows may be stale (OrchestrateInput.staleManifest from run.ts::detectStaleManifest), and the lane never passes that flag in. If a PR both moves partition_by (or a snapshot's check_cols/strategy) from inline config into yml and changes the value, a manifest compiled before the yml edit still carries the old inline value, so this returns false and the change is never reported. Thread staleManifest into the lane and skip the suppression when it is set (or compare against a base-revision manifest).
| ): boolean { | ||
| const o = before[key] | ||
| const n = after[key] | ||
| if (o === undefined || norm(o) === norm(n)) return false |
There was a problem hiding this comment.
SUGGESTION: Compare canonical values, not raw text, for inline-vs-inline changes
norm only removes whitespace, normalizes quotes and lowercases, so reordering the same dict keys ({'field': 'd', 'granularity': 'day'} vs {'granularity': 'day', 'field': 'd'}) is reported as a change, while it strips whitespace inside quoted values so 'order date' and 'orderdate' are treated as equal. Use the same canonicalization as the manifest comparison.
| if (o === undefined || norm(o) === norm(n)) return false | |
| if (o === undefined || (n !== undefined && canonValue(inlineValue(o)) === canonValue(inlineValue(n)))) return false |
| } | ||
| // Lowercase everything but string literals: `'paid'` → `'PAID'` is a change. | ||
| return s | ||
| .replace(/\s+/g, " ") |
There was a problem hiding this comment.
SUGGESTION: Whitespace inside string literals is collapsed before the literal-preserving split
.replace(/\s+/g, " ") runs over the whole SQL, so where status = 'on hold' and 'on hold' produce the same modelBody (and neither the incremental-history nor the snapshot-logic rule fires), even though the literal value changed. Split on the string literals first and collapse whitespace only in the non-literal parts. Double-quoted literals are also still lowercased, since only '…' is preserved.
| const flat = flatten(sql) | ||
| const selects = [...flat.matchAll(/\bselect\b/gi)] | ||
| if (!selects.length) return null | ||
| const head = SET_OP.test(flat) ? selects[0] : selects[selects.length - 1] |
There was a problem hiding this comment.
SUGGESTION: A parenthesized or FROM-less first branch still yields a later branch's names
flatten blanks everything inside parentheses, so for (select a as x from t) union all select b as y from u the first top-level select is the second branch and this returns ["y"] instead of the first-branch name ["x"]. When the first branch has no from, fromRel likewise crosses the set operator: select 1 as a union all select 2 as b returns ["b"]. Return null when the set operator precedes the first top-level select, and bound the first branch's list at the set operator.
| if (c === "'" || c === '"' || c === "`") { | ||
| // A doubled quote inside a string is an escaped quote. | ||
| let j = i + 1 | ||
| while (j < sql.length && !(sql[j] === c && sql[j + 1] !== c)) j += sql[j] === c ? 2 : 1 |
There was a problem hiding this comment.
SUGGESTION: Backslash-escaped quotes desync the scanner
This loop only treats doubled quotes ('') as an escape. In dialects that accept \' (MySQL, Snowflake, Redshift, Postgres E'…'), and for BigQuery '''…''' strings, the escaped quote is taken as the closing one, a later quote is treated as a new opener and the rest of the query (including real -- comments) is swallowed: select 'O\'Brien' as name, y -- note, junk yields a bogus junk column, and select 'O\'--Brien' as name, y from t blanks everything after --. Treat \ escapes (and $$…$$ / [...] quoting) as opaque while scanning.
| // Lineage returned nothing (failed, or could not analyse this side): the | ||
| // set is unknown. Filling it from the select list alone would make every | ||
| // unaliased expression look removed when only one side's lineage failed. | ||
| if (!cols.length) return new Set() |
There was a problem hiding this comment.
SUGGESTION: Empty lineage means unknown and legitimately empty, losing the literal-only case
columnLineage returns [] both on failure and when a model's output reads no source column. For a literal-only model (select 'x' as kind, 1 as one gaining current_date as load_date), both sides now return empty, so columnDelta reports nothing and selectListNames never runs — the case this feature was added for. Have the runner signal failure (null) separately so a legitimate empty result still falls through to the select-list names.
Changes since review: Review feedback commit:
Spark:pack gating, quote-aware comment stripping, effective-config (yml/project) changes not reported as removals, and lineage-less column sets treated as unknown. The earlier dialect-pack commit is reworded to match its contents.Issue for this PR
Part of #1792
Stacked on #1724 (← #1723 ← … ← #1719); merge in that order. Draft: the new rules are not yet measured on a public-repo benchmark.
Type of change
What does this PR do?
dbt incremental and snapshot semantics. Rules follow dbt-core behavior, which is the same on every adapter. The one adapter-dependent rule reads the capability table and stays silent where the behavior isn't established. Config comes from the manifest (project, yml and inline, as dbt merges them), with the head's inline
{{ config() }}on top. Every finding is advisory and never blocks.Rules
is_incremental()filter changes, and the PR doesn't say how existing rows are handledpartition_bychangeon_schema_change='fail'check_colsorstrategychanges on an existing snapshot.sqland dbt 1.9+ yml snapshotsrow_number()ordered only byloaded_at/ingestion_timestamp_fivetran_synced,_airbyte_*,_sdc_*) are not flaggedunique_key(opt-in:incrementalSliceVars)delete+insert/mergemodel filtered by a listed var (e.g. a tenant) lacks that column inunique_keyon_table_exists='replace'with an enforced contract, ordropreplace+ contract runscreate or replacetheninsert;dropremoves the table during the build. Checked against the dbt-trino 1.10.6 sourceunique_keyand strategy changes are left to the engine'sDBT011/DBT012.Also in this PR
BigQuery: …,Snowflake: …,Redshift: …,Databricks: …) fire only on their own engine once the project's engine is known. A DuckDB project was getting BigQuery partition advice.current_date as load_date). Column lineage only reports columns that read a source column.select-list.tsadds the final SELECT's own names: an item counts only when it's aliased after a complete expression or is a bare column, and*defers to lineage.Changed after an adversarial generality review, before opening. The goal was rules that fit any dbt project, not one monorepo:
How did you verify your code works?
test/altimate/review-incremental-semantics.test.tscovers the generic rules on two adapters, formatting-only edits, the opt-ins, yml snapshots and orchestrator wiring.test/altimate/review-select-list.test.tscovers the select-list names.bun typecheckis clean, and the real-world corpus is unchanged (15/15 caught, 0/5 false positives).check_colschange and adding a column underon_schema_change='fail'are reported; 0.12.10 reports neither. A model reading a tenant var with noincrementalSliceVarsconfigured stays quiet.on_table_exists: replace.Cost side effects
None. No model calls or warehouse queries.
Screenshots / recordings
N/A (CLI).
Checklist
🤖 Generated with Claude Code
Summary by cubic
Adds dbt incremental and snapshot semantics to PR review as advisory findings that fit any dbt project, following dbt-core behavior that is the same on every adapter. Config is read from the manifest (project, yml, inline, as dbt merges them) with the head's inline
{{ config() }}on top; an inline value moved to yml with the same value isn't a change, and formatting-only edits never count.Rules fire when an incremental model gains a column or its
is_incremental()filter changes without a stated plan for existing rows,partition_bychanges,on_schema_change='fail'meets added or removed columns, a snapshot'scheck_cols/strategyor query changes, or dedup orders only by a load timestamp. Per-slice runs are opt-in viaincrementalSliceVars, and each must include the slice column inunique_key. On Trino,on_table_exists='replace'with an enforced contract anddropget a suggestion because readers can see an empty or missing table. Adapter-dependent advice reads the capability table and stays silent where behavior isn't established, dialect-pack rules (includingSpark:) fire only on their own engine, and column-set rules now see literal and function-only columns (current_date as load_date) by adding the final SELECT's own names — taken from a set operation's first branch — to column lineage; when lineage returns nothing for one side, the column set stays unknown rather than reporting every expression as removed. Also fixes a false "table not found" warning against a manifest-only schema (category→kind).Written for commit 9625cfd. Summary will update on new commits.